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Abstract 
Higher education institutions utilize faculty who possess subject-matter expertise and terminal 
degrees in specific disciplines. Often, faculty are not formally trained in educational theory, 
pedagogy, or best practices related to student learning. Best practices in course design standards 
are available to faculty who seek them out. However, with little formal education or design 
traming, faculty deploy courses lacking best practices in course design to students. This creates 
a mismatch between how faculty perceive they are utilizing subject-matter expertise to educate 
students and how students receive the educational information from faculty. Before an 
investigation into whether focused training in best practices of course design standards may be 
helpful to faculty, a comparison of the course design standards faculty perceive they implement 
to the design standards that can be observed in their courses needed to be completed. 

Utilizing a quantitative approach, this study compared faculty selfperception of their 
inclusion of course design quality elements in a self-designed fully online course to investigator- 
observed course design quality elements in the faculty’s selfdesigned fully online course. 
Faculty participants in this study all taught at least 1 fully online course for a small Midwestern 
health care college during a single academic year. The results of the study demonstrated 
disconnects between faculty self-perception of the inclusion of best practices in course-design 
elements and the observed inclusion of best practice course-design elements in these areas. The 
disconnects were identified in the following areas: The presence of materials in the course that 
support efficient student course navigation and communicate general faculty expectations of 
students, the presence of materials that help students to clearly understand the syllabus and what 
work is expected to be completed within a certain timeframe, and the presence of clear 


information about the delivery method of the course. 
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Chapter 1 
Elements of Quality Course Design 

Backward Design is a method of curriculum creation described by Wiggins and McTighe 
(1998) that helps faculty create course materials and experiences to meet the learning and 
knowledge needs and desires of students. According to Wiggins and McTighe, the creation of 
courses with the end in mind identifies standards to direct and formulate the materials and 
activities utilized by instructors in a course. Providing a framework for identification of materials 
and strategies, Backward Design supports the construction of learning experiences for students, 
guiding them to demonstrate necessary achievement at the end of a course (Wiggins & McTighe, 
1998). This research clearly articulated that intentional learning experiences by students must be 
deliberately designed by mstructors. 
Background of the Problem 

Backward Design is not an intuitive manner of course planning. Wiggins and McTighe 
(2005) pointed out in Understanding by Design that instructors often perceive this method of 
course planning to be uncomfortable, as it detours from a traditional planning methodology of 
listing course activities without directly relating the activities to the course outcomes. Traditional 
planning methods are driven by course activities and breadth of content rather than identification 
of the necessary materials and activities that allow students to perform course expectations 
(McTighe & Wiggins, 2012). Support in the utilization of alternate, student-performance-based 
planning methods must be given to those responsible for course planning. 

The authors of Understanding by Design partnered with the Association for Supervision 
and Curriculum Development to create a guide sheet to assist educators in implementing the 


Backward Design process (Wiggins & McTighe, 2005). Similarly, Wiggins (2012) collaborated 
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with the University of North Carolna at Greensboro to provide a framework for embedding 
Backward Design into faculty processes. A curriculum planning process directed toward nurse 
educators was presented by Emory (2014). Likewise, Michael and Libarkin (2016) offered a 
similar course- development framework that embedded Backward Design into course 
development for higher education faculty in all content areas. These articles have provided a 
foundation for faculty to develop courses aimed at the intentional integration of learning 
outcomes in course work at the lesson or unit level. The Backward Design framework also 
directed the intent of this study to expand outcomes-focused design into the development ofa 
fully online course. 

Although Backward Design provides faculty with a framework for mtentional course 
design, faculty have the freedom to deviate ftom the design process. Ensuring that online course 
materials meet a mmimum of standards, the Quality Matters organization developed a standards 
measurement tool Quality Matters, an organization that expanded from the Maryland Online 
Consortium, provides training, a rubric, and supporting materials for higher education 
institutions to evaluate online course design. Quality Matters methods are used specifically to 
evaluate eight different categories related to the design of a course. These categories are the 
following: course overview and introduction, learning objectives or competencies, assessment 
and measurement, instructional materials, learning activities and learner interaction, course 
technology, learner support, and accessibility and usability (Helping You Deliver on Your 
Online Promise, 1997). Similarly, the Online Learnng Consortium has endorsed the Open 
SUNY Course Quality Review (OSCQR) rubric for online course review in higher education. 
The OSCQR evaluates the design of the following areas of a course: overview and mnformation, 


technology and tools, design and layout, content and activities, student-student or student-faculty 
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interactions, and assessment and feedback (OSCQR, n.d.). Both tools can be utilized by faculty, 
designers, or course reviewers to evaluate the comprehensiveness and quality of online course 
design. 

Tools and supports such as the Backwards Design, Quality Matters, and the OSCQR are 
in place to assist faculty in developing courses that promote student interaction and achievement 
of course outcomes; yet, many faculty continue to perceive their own design abilities to be 
content-centric (You, 2010). Institutions of higher education traditionally place more value on 
research than teaching excellence; therefore, faculty who are traditionally research focused rather 
than instruction focused need structured supports to create effective educational experiences for 
students (Marentic Pozarnik & Lavric, 2015). Consequentially, Marentic Pozarnik and Lavric 
(2015) pointed out that providng exemplary targeted trainmg in teaching and learning is 
especially important for beginning higher education faculty, as it establishes standards and 
practices to support the teaching role 

In the article, “Accidental Composition: How the Ph.D. Machine Fails Our Students,” 
Vance (2018) discussed the negative impacts on students when a faculty member does not know 
how to teach. Vance also pomted out that faculty hiring committees should be expected to 
identify any pedagogical training needs to the faculty chair so that the individual faculty 
member’s training needs can be addressed during the orientation or on-boarding process. 
Institutions of higher education fail the faculty, and therefore the students, by neglecting an 
environment of continuous pedagogical improvement, resulting in faculty who either cannot 
teach or who are left to their own devices to learn or relearn how to teach (Vance, 2018). 

Formalized pedagogical training and implementation of standardized course design 


support tools are lacking in higher education (Khalil & Elkhider, 2016). Without organized 
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pedagogical supports such as training faculty to teach in higher education, faculty default to 
creating courses that are comfortable for them to teach, rather than searching out ways for 
students to engage fully with the learning process (Negassa & Engdasew, 2017). The lack of 
intentional faculty support with the tools and training to provide quality content and pedagogy is 
a failure in institutions of higher education. 

Statement of the Problem 

Although online course design tools and evaluation methods are available, no 
standardized process exists to guide faculty through the how of implementation. Wiggins and 
McTighe (1998) and Khalil and Elkhider (2016) worked from the assumption that those 
implementing Backward Design are formally trained educators. Michael and Libarkin (2016) 
pointed out that the “lack of pedagogical training that faculty members receive becomes apparent 
when a new instructor has to teach a course for the first time” (p. 46). An assumption of Wiggins 
and McTighe and Khalil and Elkhider was that all educators have at least a working 
understanding of pedagogy. Michael and Libarkin found that higher education faculty lack 
pedagogical training results in outcome misalignment and a lack of student-centered interaction 
in course design (You, 2010). 

Formalized design processes and alignment of learning expectations to educational 
activities remain challenges for most faculty who have not been formally trained in pedagogical 
practices (Shaver, 2017). Faculty in higher education need support to create intentional 
connections among content, assignments, and course outcomes. Recognition by faculty of the 
need for intentional design to create course material and course outcome connections is 
necessary but cannot occur without training. Research of Shaver (2017) has shown the 


importance of sound pedagogical and course design practices in higher education. However, the 
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degree to which faculty perceive their own course design ability has not been measured in the 
literature. 
Purpose of the Study 

The purpose of this study was to investigate if higher education faculty accurately 
perceive the extent to which they are providing students with course materials that meet 
minimum design standards. This was determined through a comparison of the faculty’s self 
perceived ability to design a quality course and a course evaluation using the design standards of 
the OSCQRrubric. The comparison identified gaps between faculty’s selfperceived ability to 
design quality courses and observed inclusion of best practice elements in deployed fully online 
courses. 

The focus of the study was on the design of online courses at a small, private, health care 
college. Faculty were surveyed to determine their perception of their own competency in the 
design of welcome and introductory information in courses at both the undergraduate and 
graduate levels. The perceptions were compared with an evaluation of the course based on the 
OSCQRrubric’s Overview and Information subscale to determine if any gaps existed between 
faculty’s self-perception of the inclusion of essential quality elements and the existence of those 
elements in the deployed course. 

Importance of the Study 

According to Michael and Libarkin (2016), higher education faculty typically have no 
formalized training in andragogical methods or course design quality. Support for faculty to 
implement best practices in course design and pedagogy 1s necessary to create overt connections 
among course materials, assignments, and course outcomes (Khalil & Elkhider, 2016). 


Facilitating acquisition of new knowledge and creating a cognitive structure around the new 
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knowledge is necessary for students to learn successfully (Charikova & Zhadanov, 2017). The 
OSCQR tool is intended to evaluate implementation of the essential elements of course design 
necessary to elicit student performance that meets the intended course outcomes. Comparing the 
faculty survey and the OSCQRscores of courses taught by the surveyed faculty determined how 
selfaware and adept faculty were in developing course materials and assignments overtly 
connected to the course outcomes. This study also identified additional traming opportunities for 
faculty regarding how to incorporate course design elements that assist students in meeting the 
intended outcomes of a course. 
Definition of Terms 

Assessment: evaluation or ability of a student to demonstrate the performance of 
academic learning. 

Assignment: work assigned as a part of academic study. 

Backward Design: method of course design, described by Wiggins and McTighe (2005) 
that identifies the intended outcomes prior to the creation of any course activities. 

Case Study: research process related to the implementation of a newly designed tool. 

Course Design: a process completed by an instructor or mstructional team that plans for 
successful completion of academic study for students. 

Course Material: activity, reading, interaction, lecture, or other content used to support 
student achievement and demonstration of course outcomes. 

Course Outcome: statement that describes what a student is expected to demonstrate 
after fully interacting with course materials. 

Curriculum: a program of study. 


Faculty: higher education instructor. 
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Higher Education: college or university instructional program. 

OSCOR Rubric: a tool designed by SUNY and the Online Learning Consortium that 
informs faculty or instructional design teams of the quality, effectiveness, and efficiency of 
course design. 

Pedagogy. processes and methods used to teach an academic topic. 

Limitations and Delimitations of the Study 

The researcher conducting this study assumed that the faculty member participants did 
not already intentionally utilize design principles and had not already utilized instructional 
designer assistance when developing a course. However, faculty who participated in this study 
may have already been exposed to the elements of quality design as measured by the OSCQR 
rubric. The study was designed to review a variety of course content areas at multiple levels of 
instruction taught by different faculty. The study’s breadth was intended to gain insight into the 
varying course design abilities and perceptions of multiple faculty members at one small college. 
Essential Questions and Objectives 

The main objective of the study, to identify faculty perception of their own ability to 
provide students with course materials that meet minimum design standards, was investigated 
through the question: Do faculty accurately perceive their course design ability? To meet this 
objective of the study, courses designed by individual faculty members were scored to determine 
whether their course design met minimum standards on the OSCQR Overview and Information 
subscale. 

Two additional objectives of this study were to identify faculty strengths and weaknesses 
(through a comparison of faculty perception of course design abilities of course overview and 


information and scores on the OSCQR Overview and Information subscale) and to identify areas 
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for suggested faculty training in course design standards. These last two objectives attempted to 


answer the question: Could training be provided to faculty to address potential gap areas? Table 


1 summarizes the objectives and methods of measurement for each objective in this study. 


Table 1 


Table of Study Objectives and Measurement Methods 


Objective 


Measure faculty perception of their own 
course design ability 


Determine overall course design quality 
based on the minimum standards of the 
OSCQR Overview and Information 


subscale 


Identify faculty strengths and weaknesses 
through a comparison of faculty perception 
of course design abilities with scores from 
the OSCQR Overview and Information 
subscale 


Identify areas for suggested faculty training 
in course design standards 


Measurement Method 


An eight-statement survey based on the best 
practices survey utilized in, A Study of Faculty 
Members’ Perceived Utilization of Best 
Practices in Distance Learning Course 
Design and Delivery and the Role of 
Instructional Designers by You (2010) was 
given to faculty for self-evaluation. 


Eight items from the OSCQR rubric’s 
Overview and Information subscale were 
scored by the investigator to evaluate the 
course design of courses developed by 
surveyed faculty, resulting rubric output 
indicated the estimated total additional design 
workload needed to meet minimum rubric 
standards. 


Faculty survey results were compared with 
OSCQR Overview and Information subscale 
scores. 


The OSCQR subscale scores were reviewed to 
identify patterns (1e., in any areas with 2+ 
hours of additional design work necessary to 
meet minimum rubric standards). 


Note. Study objectives with corresponding measurement methods 


Summary 


Faculty in higher education are more often discipline-specific subject matter experts 
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rather than educational practice experts. The research noted in this review demonstrated the need 
for faculty support in the design of teaching strategies in their deployed courses. Through a 
survey of faculty selfperceptions of about course design best practices, this study identified the 
degree to which faculty perceive their own course design ability. Through the comparison of 
faculty perception of course design abilities with the OSCQR subscale, this study sought to 


identify areas for potential design concepts support or training. 
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Chapter 2: 
Review of Literature 

The Elementary and Secondary Education Act of 1965 and subsequent reauthorizations 
were the driving forces for educational accountability in K-12 education. Following the act’s 
passage, research about which teaching and learning strategies provided the most impact on K-12 
student learning began to emerge. As students progressed from the K-12 educational 
environment into higher education, they began to expect the same quality of teaching and 
learning to be offered as a part of their higher education experience. Such an expectation by 
students generated the need to improve the educational accountability of higher education. To 
ensure educational accountability, design methods and evaluation tools were needed so faculty 
could measure the effectiveness of delivered course content. The focus of this literature review is 
to discuss research demonstrating faculty strengths and weaknesses in providing students with 
course materials that integrate best practices in course design. 
History 

The importance of how education is delivered and the connections between educational 
delivery and student achievement were noted as early as the 1970s. In his article, “Five 
Evaluation Frameworks: Implications for Decision Making in Higher Education,” Gardner 
(1977) set out to provide guidance in how to best measure educational accountability in terms of 
delivery and achievement. According to Gardner, student evaluation methods, teaching 
strategies, and educational methods should all be considered when determming educational 
effectiveness. That is, appropriate pedagogical methods for learning outcomes are necessary to 
verify student achievement of course outcomes. 


Looking forward to the 1990s, research was published about the effectiveness of course 
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delivery in higher education. Ramsden (1991) concluded that students are uniquely qualified to 
judge instructional effectiveness without any formalized traming. As consumers of education, 
students are able to measure objectively teaching performance simply because they see so much 
instruction. A student’s interpretation of teaching performance is based on the instructor’s ability 
to convey clearly expert knowledge to the student through effective course design strategies 
(Ramsden, 1991). 

Liow, Betts, and Lit (1993) identified a direct relationship between teaching methods and 
learning outcomes and emphasized the importance of intentional planning of such methods in 
order to reach outcomes. Up until this period, no resources were found that addressed the 
relationship between intentional course design and learning outcomes. The authors concluded 
that in order for students to have the greatest opportunity to achieve all the educational outcomes 
for learnng, courses must be developed with applicability to the expected learning goal rather 
than to a specific group of students or toward specific content consumption. However, Liow et 
al. concluded there is no correlation between the implementation ofa specific teaching strategy 
and increasing student performance of specific learning outcomes. That is, the student 
demonstration of learning outcome achievement relates to the learning design intentionality 
rather than the learning method (Liow et al, 1993). Therefore, based on the work of Liow etal., 
student performance can be directly related to how clearly the materials are delivered rather than 
the strategy used to deliver the materials. 

As a result of educational research in the 1990s, learning came to be understood as a 
process rather than a culminating event. For example, Lea (2004) discussed the relationships 
among learning, the method of learning, and the formation of knowledge. Lea identified this 


process as a negotiation that is unique for each student in collaboration with the teacher, the 
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materials, and the environment. Each experience 1s different and must be intentionally planned. 
Deliberate design of instructional interactions and the learning environment must be uniquely 
addressed from both the instructor and student perspective. 

Recognition of the need to consider the student experience in terms of the educational 
process toward the achievement of learning outcomes was brought to the forefront in the 
Kelting-Gibson (2005) study, “Comparison of Curriculum Development Practices.” The data 
results revealed that students who participated in a course that was developed using a Backward 
Design process outperformed students who participated in a course that was developed using a 
traditional design process. Using design standards to create intentional design within online 
courses provides students with a structure that supports learning in the upper domains of 
cognition as defined by Bloom’s Taxonomy (Dole & Bloom, 2009). Educational settings that are 
centered on learning, knowledge, assessment, and community solicit greater student learning 
outcome achievement than do educational settings that are constructed in a content-centered 
manner (Dole & Bloom 2009). 

Simply placing content in front of students for them to sift through in a learnng 
management system is not enough to engage students in authentic learnng (Hixon, Barczyk, 
Buckenmeyer, & Feldman, 2011). Studies relating the importance of the design of learning and 
learnng outcomes eventually led the process of Backward Design created by Wiggins and 
McTighe (1998). Their work provided the first comprehensive design guide for educators, 
focusing on content, instructional strategies, and student achievement measurements. Although 
not written for any particular educational level, the book provided guiding principles for 
intentional instructional and materials design with the goal of learning for student outcomes. 


Wiggins and McTighe reiterated, frequently, the need to consider continually the student and 
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supports necessary for student achievement, rather than the needs of the educator. All focus 
should be on the student and how to guide the student to achieve the desired results of the course 
using targeted performance measures. The basic foundation of the Wiggins and McTighe design 
guide is to consider what the student should know, understand, and do at the conclusion of the 
course. The end of the course is intended to be the beginning of the planning process (Wiggins & 
McTighe, 2008). Implementing the Backward Design process can specifically craft a learning 
environment conducive to student achievement of expected learning outcomes through the 
utilization of appropriate instructional strategies and materials. 

Even with the guidance provided by Backward Design concepts and processes, research 
continued in fields related to best practices for instruction and design, particularly in online 
instructional best practices. Through best practice research, an organization grew from grassroots 
efforts in Maryland, eventually leading to the creation of Quality Matters. Quality Matters (QM) 
directly supports implementation of design and interaction standards that fosters student learning 
(Helping You Deliver On Your Online Promise, 1997). Research then began to include design 
expectations as outlined by QM, resulting in much more evidence about how online instruction 
should be implemented. Varonis (2014) studied the processes outlined by QM, pointing out a 
disconnect between subject matter experts’ instructional practices and best practice instruction 
and design. Faculty, the subject matter experts, do not typically possess the pedagogy or strategy 
knowledge to support the resources students need in a fully online environment (Varonis, 2014). 
The literature referenced in this section is specific to lesson or unit planning for achievement of 
learning outcomes. 

Many faculty in higher education lack educational training or experience (Chao, Saj, & 


Hamilton, 2010). As a result, the deficit knowledge and skills held by faculty in the intentional 
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creation ofa learning environment that solicits authentic learning by students, is naturally 
inclusive of student individuality and conveys clearly stated outcomes and expectations (Smidt, 
McDyre, Bunk, Li, & Gatenby, 2014). Therefore, faculty need intentional guidelines to navigate 
the course creation process (Chao et al., 2010). Equally helpful in the design process 1s the 
collaboration between the faculty and an instructional designer (Chao et al, 2010). Not all 
faculty have unlimited access to instructional designers, but access to faculty who have 
successful experience in course design is nearly as helpful as the guidance of an instructional 
designer (Chao et al., 2010). 

Research has not only focused on what formally trained educators deem necessary as best 
practices of quality course design but also on the student viewpoint. Fayer (2014) noted that 
students identified course organization, relevance of course work to authentic experiences, 
timely feedback from the faculty, and the ability to self-assess achievement of learning outcomes 
as necessary elements in successful courses (Fayer, 2014). These course elements, as identified 
by students, are aligned with the best practice elements identified by the Backward Design 
process, QM, and the relationship between teachng methods and student achievement (Helping 
You Deliver On Your Online Promise, 1997; Liow et al., 1993; Wiggins & McTighe, 2008). 

To date, the literature documents the practice of Backward Design beginning with 
Wiggins and McTighe (1998) through present day (Chao et al., 2010; Jones, Vermette, & Jones, 
2009; Khalil, & Elkhider, 2016; Kelting-Gibson, 2005; Lea, 2004; Lenert, & Janes, 2017; 
Marshall, 2015; Michael & Libarkin, 2016; Neal & Hampton, 2016; Reynolds & Dowell Kearns, 
2017; Sun & de la Rosa, 2015; Vance, 2018; Whitehouse, 2014). Additionally, Marshall (2015) 
and You (2010) presented research that called out the lack of pedagogical training for higher 


education faculty and characterized faculty’s individual implementation of intentional 
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pedagogical practices into courses. However, the topic of faculty integration compared to faculty 
perception of pedagogical practices into course design is an understudied phenomenon. 
Current Implications of Literature 

Research and writings related to the Understanding by Design tenets brought about 
research related to the implementation of the concepts of Backwards Design (Chao et al., 2010; 
Jones et al, 2009; Khalil, & Elkhider, 2016; Kelting-Gibson, 2005; Lea, 2004; Lenert & Janes, 
2017; Marshall, 2015; Michael & Libarkin, 2016; Neal & Hampton, 2016; Reynolds & Dowell 
Kearns, 2017; Sun & de la Rosa, 2015; Vance, 2018; Whitehouse, 2014). This work resulted in a 
more defined course development framework intended to provide guidance for faculty in higher 
education. Faculty guidance regarding course design is necessary because, as pomted out in 
several studies, higher education faculty have not been formally trained in the pedagogical 
practice application necessary for facilitating student learning (Negassa & Engdasew, 2017). 
According to Wiggins and McTighe (1998) instructors are encouraged to capitalize upon the 
resources available to them to identify, write, design, and deliver courses that guide students 
toward the achievement of the identified learning outcomes. Subsequently, much effort has gone 
into identifying and refining a process that supports content creation and student course work 
expectations that faculty can utilize when planning instruction intended to produce effectively 
student learning outcomes (examples: Jones et al, 2009; Whitehouse, 2014). 

Research conducted between 1998 and present day has advanced a process of course 
development with emphasis on Backward Design in higher education. Much of this work has 
extended educational theory long embedded in K-12 education to higher education. Discipline- 
specific studies by Michael and Libarkin (2016) and Neal and Hampton (2016) focused on 


implementation of Backward Design in higher education. Although these studies focused on 
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differing methods, educational levels, and content areas, both studies concluded that Backward 
Design facilitates the internalization and application of the intended learning outcomes of 
formalized education. 

Faculty often lack the experience and training to recognize their own gaps in pedagogical 
practices (You, 2010). Institutions of higher education often fail the faculty and, therefore, the 
students by placing more focus on research practices than on pedagogical improvement (Khalil 
& Elkhider, 2016; Vance, 2018). The focus on research rather than instruction results in faculty 
relyng on implementation of the instructional methodologies and tools with which they are most 
familiar (You, 2010). Faculty need to be empowered to identify, discuss, and engage in 
pedagogical practices that are unfamiliar to them in order for learnmg mastery to occur for 
students (Brown & Ramasamy, 2017). In order for quality courses to be deployed, institutions of 
higher education must devote resources toward course design processes that are strongly rooted 
in pedagogical strategies and technical supports (Rucker, Edwards, & Frass, 2015). Institutions 
of higher education need to have professionally tramed support personnel and easily accessible 
course design and review processes for faculty in order to mplement effectively best practices in 
online instruction. 

Faculty need to have access to the tools and processes of informed instruction, including 
the ability to make data-driven decisions regarding instruction, in order to provide the most 
appropriate learning environment for students. “Reflection on practice, making meaning 
together, and sharing of expertise are essential for those navigating the unfamiliar landscape of 
online teaching and learning” (Collay, 2018, p. 35), and there is a “critical need for “good 
modeling’ for future instructors, within the quality design process” (Lenert & Janes, 2017, p. 10). 


Faculty should have access to pedagogical and design trainng related to quality course design, 
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guidance from instructors who have successful design experience, or feedback from instructional 
designers in order to create a quality course design with learning and outcomes at the forefront 
(Michael & Libarkin, 2016). Recognition of the need for inclusivity specific to a group of 
students in a class or the students’ educational community is a learned trait by faculty and must 
be fostered in order to be effectively learned (Lenert & Janes, 2017). Faculty need to be flexible 
in adapting to an environment of planning, growth, change, continuous improvement based on 
student feedback, uncertainty, and revision of course content and instruction (O’Connor, 2012). 
When faculty recognize the positive impact that refining content, pedagogical practices, and 
assessments can have on student success, they will be more likely to embrace the opportunity to 
learn and incorporate such practices (Collay, 2018). Course review processes that identify gaps 
in instructional knowledge also need to be in place to offer faculty development that expands the 
teaching capabilities of the online instructor (Northcote, Gosselin, Reynaud, Kilgour, & 
Anderson, 2015). 

Adaptability as outlined by O’Connor (2012) and Lenert and Janes (2017)(Source not 
listed in reference section.) can be supported through the identification of gaps in course content 
via tools such as the QM and Online Learning Consortium standards. Faculty who participate in 
QM training are informed in areas such as writing and associating learning objectives and 
creating outcome assessments (Sun & de la Rosa, 2015). Research has demonstrated that courses 
designed by QM-trained faculty produced an increase in student learning from instructional 
materials, learner interactions, and course technologies (Sun & de la Rosa, 2015). Although all of 
these identified areas are impacted by faculty traming, none is more improved than student 
interaction (Sun & de la Rosa, 2015). 


Though the literature has demonstrated the necessity of faculty training, it is also 
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important to determine baseline information for how faculty are performing and for how they 
perceive their own performance related to online course design. Marshall (2015) focused on the 
connection between faculty rank and course design to determine if faculty who held a higher 
faculty rank in thei institution were more able to provide a course that utilized quality course 
design practices. The study found that no connection was evident between faculty rank and 
quality course design (Marshall, 2015). However, there continues to be a gap in the literature as 
there has not been a comparison of faculty self-perception of course design quality with course 
design quality standards. Without such measures, it is difficult to level appropriately the needed 
supports and training to build on faculty strengths, illuminate misconceptions, or identify areas 
of needed improvement. 
Summary 

Institutions of higher education continue to work to identify the best methodologies for 
implementing intentional instructional design for faculty who lack formal educational training. 
The studies discussed in this chapter focused on faculty traming and design expertise. No study 
was identified by the researcher that identified alignment between student-centered instructional 
design and deployed instruction in the higher education setting. As the literature review 
discussed, a study focused on the alignment of faculty selfperception as compared to the course 


materials provided for students in fully online courses is needed. 
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Chapter 3 
Methods 

Methodology 

This study surveyed faculty who had taught at least one fully online course to determine 
faculty self-perception of course design quality within the introductory components of the 
course. The survey was based on You’s (2010) Likert-scale rating of self-perception statements, 
resulting in numerical values associated with the survey responses. Numerical values were also 
obtained from the OSCQR Overview and Information subscale. The OSCQR rubric is a course 
review tool made available by Online Learning Consortium, in association with the State 
University of New York (SUNY). The scores were the result of reviews completed by the 
researcher who is an experienced educator with 12 years of experience in instructional design. 
The OSCQR subscale values reflected either that the course had met the OSCQR standard or that 
there would be a specific amount of workload associated with bringing the course elements up to 
the standard for each criterion. These numerical measurements were analyzed and compared, 
resulting in a quantitative study. The study was designed to be utilized in a small private health 
care college, limiting the number of participants. 
Tools 

As mentioned, two tools were utilized in this study. First, the faculty self-perception 
statements originated from You’s (2010) work A Study of Faculty Members’ Perceived 
Utilization of Best Practices in Distance Learning Course Design and Delivery and the Role of 
Instructional Designers. The full list of statements from You’s study, based on a forced-choice, 
four-degree Likert scale, are provided in APPENDIX A. Additional optional questions were 


asked at the conclusion of the survey to gather respondent demographic information. These 
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questions are provided at the end of APPENDIX A. 

Second, the OSCQR rubric was used to evaluate the course designs. As displayed in 
APPENDIX B, the OSCQR was constructed to rate the design of a course in the areas of 
overview and information, technology and tools, design and layout, content and activities, 
interaction, and assessment and feedback (OSCOQR, n.d.). The OSCQR identified the total 
number of work hours, post design, each course would need in order to meet the minimum 
qualifications of a well-designed course. 

Data Collection 

Essential questions answered in this study were: 

e Do faculty accurately perceive their online course design abilities? 

e@ What gaps exist between faculty’s self-perception of online course design and 

implementation of essential course design elements in their courses? 

e Could training be provided to faculty to address potential gap areas? 

Content validation data collection. Data collection was conducted in two phases. First, 
content validation data was collected to determine alignment between faculty self-perception 
statements and the OSCQR rubric subscales. The participant sample for content validation 
consisted of one instructional designer and one faculty member per rating iteration. Instructional 
designer participants must have had a minimum of two years of online design experience and 
faculty member participants must have earned a terminal degree and had experience teaching 
fully online courses. Content validation participants must also have had design or teaching 
experience in a Midwestern health care-focused institution. Participants ftom the investigator’s 
home institution and the institution under investigation were excluded. 


To initiate the validation, statements were selected from You’s (2010) 30-statement 
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survey (APPENDIX A) that aligned with the selected items from the OSCQRrubric’s seven 
subscales (APPENDIX B). The researcher paired a single selfperception statement from the 
You study with a single criterion from the OSCQRrubric. Once results of each pair surveyed 
were completed, the researcher edited the self-perception statement to incorporate the feedback 
provided by the participants. The iterative process resulted in modified selfperception 
statements that were subsequently provided to a new pair of participants for feedback. New 
participants were identified and surveyed until items were determined to be valid for alignment 
of measures, clarity, simplicity, and ambiguity. 

The initial goal was to align all 30 of You’s (2010) faculty selfperception statements 
with all seven of the OSCQRrubric subscales. However, as described in Chapter 4, the only 
complete OSCQR subscale for which valid alignment could be determined was the OSCQR 
Overview and Information subscale, which corresponded to eight faculty self-perception 
statements related to course welcome and introduction information. 

Faculty self-perception and OSCQR data collection. Once validation was complete, 
data collection for the study commenced. Faculty members who had taught at least one fully 
online course were given a randomized participant number and the eight-statement electronic 
survey. The survey provided numeric representations of faculty members’ qualitative perceptions 
of the integration of course design best practice into the course welcome and introductory 
information. A research assistant was utilized in the assignment of the participant numbers in 
order to randomize the participants. 

The faculty also selected a fully online course that they had developed in the 2018-2019 
academic year and communicated their selection to the research assistant, who identified the 


course by the same randomized number used on the faculty survey so that the results could be 
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matched. The researcher blindly reviewed the identified course using the OSCQR Overview and 
Information subscale, which includes information related to the general setup of the course and 
clear expectations for participation and learnng. This measurement provided quantitative data to 
determine the quality of each course’s design on the indicated subscale of the OSCQR rubric. 
The ultimate goal of using You’s (2010) statements and the OSCQR rubric was to compare 
faculty selfperception of design quality with the level of design quality identified by an external 
reviewer using a standardized tool. 
Reliability 

The reliability of You’s (2010) study statements was determined by implementation of 
the Rasch model. The statements were determined to have a 0.83 reliability per participant of the 
36-participant group, indicating the questions in the initial study were constructed and delivered 
in a reliable manner. Because this study was based on You’s model and statements, a reasonable 
determination was made that reliability did not need to be piloted again in this study. 
Validity 

You’s (2010) study statements had already been determined to be reliable for content and 
construct as apart of the initial study. As a part of the pilot study included in You’s (2010) 
research, the statements were provided to subject matter experts in instructional design to 
determine the construct of the statement and the appropriateness of the language. The statements 
were then modified and presented to 70 faculty members at two separate universities who were 
chosen through a stratified random sampling method. The methods used by You validated the 
statements’ content and construct. 

The OSCQRrubric was developed and tested by a broad range of instructional designers, 


librarians, distance learning directors, and technologists to create criteria and a rating system that 
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were based on the best practices of teaching and learning in higher education. This rubric was 
tested in the SUNY system and found to be a valid instrument in determining course design with 
alignment to the Open SUNY fundamental competencies for online teaching (OSCQR, n.d.). To 
ensure alignment between the survey statements and the OSCQRrubric, the researcher 
conducted a content validation index. 

Data Analysis 

Content validation data analysis. The content validation data was analyzed by 
calculating the mean response value of the pair of respondents in each round. Validity was 
considered to be reached when item scores were greater than 0.75. Full alignment for a complete 
subscale of the OSCQR rubric was necessary to ensure validity and reliability of that OSCQR 
subscale. In other words, if alignment was obtained between some of the self-perception 
statements and some, but not all, of the criteria within an OSCQR subscale, the scale was not 
utilized because of validity concerns. Because of this validation requirement, the content 
validation process only resulted in a full alignment between the faculty self-perception 
statements and the OSCQR Overview and Information subscale (these results are presented in 
detail in Chapter 4). 

Faculty self-perception and OSCQR data analysis. In order to analyze the study data, 
each of the eight validated survey statements was paired with an OSCQR Overview and 
Information subscale criteria. The mean score of each pair was calculated to determine an 
average score for each survey participant’s question group. A Wilcoxon Signed Ranks Test was 
used to analyze the paired responses. This nonparametric measurement identifies the differences 
between individual responses of paired data to determine the averages of the differences between 


individual pairs. The Wilcoxon Signed Ranks Test was additionally used to determme if the two 
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dependent data sets could be assumed to have similar distributions applicable to all populations. 
The OSCQR Overview and Information subscale also resulted in a mean score. The mean scores 
were then used to calculate t-values, which reflected the difference between the rubric scores and 
survey scores, and P-values, to identify the probability that the t-values were achieved by 
chance. These values demonstrated associations between faculty perception of the 
implementation of quality course design and scored course design quality. 
Summary 

The quantitative methodology used by the researcher in this study aligned selected survey 
statements from You’s (2010) study to the OSCQR rubric’s Overview and Information section. 
Items included in the data collection were deemed to be reliable and validated through a content 
validation process that resulted in a reduced usage of the self-perception statements and the 
OSCQRrubric. The alignment between the faculty survey statements and the OSCQR scores 
was analyzed to determine differences between faculty self-perception of course design 


implementation and the OSCQR rubric value associated with courses designed. 
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Chapter 4 
Findings 

Two data collection processes were initiated in the completion of this study. One data 
collection process was to validate the content alignment of the faculty self-perception statements 
with the OSCQR rubric criterion. The second data collection process was to survey the faculty to 
determine their selfperceptions of the inclusion of quality course design elements as identified 
in the OSCQR Course Overview and Information subscale. This section describes the findings 
from each of data collection process. 

Content Validation Results 

The purpose of the content validation was to ensure alignment between the self- 
perception statements and the OSCQRrubric criterion. A total of three designer-faculty pairs 
were involved in the iterative validation process described in Chapter 3. One faculty member 
chose to withdraw from the content validation process once the process was initiated, resulting in 
three course designers and two faculty members providing their input to complete the content 
validation process. 

One pair at a time, an instructional designer and a faculty member were asked to provide 
feedback that continued until consensus was reached. The completion of the data collection 
resulted in the inability to ensure validation among the 30 faculty self-perception statements 
from You’s (2010) survey and the criterion in all seven OSCQR rubric subscales. Content 
validity results displayed in Table 2 showed that not all of You’s survey questions could be 
validated for alignment of measures, clarity, simplicity, and ambiguity with > 0.75 index value. 
Table 2 


Faculty Self-Perception Content Validity Findings 
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Overview & Information* 


Learners have easy access to a 
well-designed and up-to-date 
gradebook. 


An orientation or overview is 
provided for the course overall, as 
well as in each module. Learners 
know how to navigate and what 
tasks are due. 


A printable syllabus is available to 
learners. 


Course includes a Course 
Information area that deconstructs 
the syllabus for learners in a clear 
and navigatable way. 


Course includes links to relevant 
campus policies on plagiarism, 

computer use, filing grievances, 
accommodating disabilities, etc. 


Course information states whether 
the course is fully online, blended, 
or web-enhanced. 


Course objectives—outcomes are 
clearly defined, measurable, and 
aligned to learning activities and 
assessments. 


Course provides contact 
information for mstructor, 
department, and program. 


Technology & Tools* 


Technical skills required for 
participation in course learning 
activities scaffold in a timely 
manner (orientation, practice, and 
application-where appropriate). 
Design & Layout* 


A logical, consistent, and 
uncluttered layout is established. 
The course is easy to navigate 
(consistent color scheme and icon 


Self-Perception Statement? 


I make sure the gradebook is 
current and understandable by 
students. 


I provide course navigation 
instructions, as well as course, 
module, or unit introductions and 
due dates so students know what is 
expected and how to meet the 
expectations. 


I provide students with a viewable 
& printable syllabus. 


I distribute assignment-task due 
dates throughout the semester, 
explain the patterns of due dates, 
and provide easy navigation to 
assignments. 


I reference relevant College or 
University policies in the syllabus 
or in the course. 


I clearly state on the homepage that 
the course is in an online format. 


I use measurable learning 
objectives to design student 
assessments. 


I provide students with my contact 
information and office hours. 


SelfPerception Statement? 


I pay particular attention to the 
skills required to successfully 
complete learning activities and 
scaffold technical skills when 
necessary. 

Self-Perception Statement? 

I make sure the look of my course 
is uncluttered and easy to navigate. 
I make sure the course has a logical 
and consistent layout. 


26 


Content Validity Index 
0.927* 


0.781* 


0.865* 


0.823* 


0.917* 


0.990* 


0.906* 


0.979* 
Content Validity Index 


0.760 


Content Validity Index 
0.833 
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layout, related content organized 
together, self-evident titles). 


Content & Activities? 


Self-Perception Statement? 


Course offers access to a variety of I use a variety of materials, 


engaging resources that facilitate 
communication and collaboration, 
deliver content, and support 
learnng and engagement. 


Course provides activities for 
learners to develop higher-order 
thinking and problem-solving 
skills, such as critical reflection 
and analysis. 


Course provides activities that 
emulate real-world applications of 
the discipline, such as experiential 
learning, case studies, and 
problem-based activities. 


Where available, Open 
Educational Resources, free, or 
low-cost materials are used. 


Text content is available in an 
easily accessed format, preferably 
HTML. All text content is readable 
by assistive technology, including 
a PDF or any text contained in an 
image 

A text equivalent for every nontext 
element is provided (alt tags, 
captions, transcripts, etc.). 

Text, graphics, and images are 
understandable when viewed 
without color. Text should be used 
as a primary method for delivering 
information 


Interaction? 


Expectations for timely and 
regular feedback from the 


including different media formats, 
to present course materials in order 
to facilitate student engagement 
(e.g., web pages, audio, or video 
clips). 

I use active learning activities (e.g. 
discussions, case studies, problem- 
based learning, and simulations 
etc.) to develop students’ higher- 
order thinking skills such as 
problem solving, critical reflection, 
and analysis. 


I use simulated or real-world 
problems (e.g. case-studies, 
problen-based learning activities) 
to engage students’ learning. 


I provide students with free or low- 
cost materials whenever possible. 


I strive to create an inclusive and 
accessible course environment 
through the design of my content 
and activities. 


I provide a text alternative for all 
images and closed captions for all 
videos. 


The visual aids I use in the course 
can be clearly understood without 
color and are not used as the only 
method of delivering information 
to students. 


SelfPerception Statement? 


I tell my students when they can 
expect to receive feedback on 


Zi 


Content Validity Index 
0.823 


0.823 


0.917 


0.938 


0.708 


0.927 


0.490 


Content Validity Index 
0.750 
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instructor are clearly stated 
(questions, e-mail, assignments). 


Expectations for interaction are 
clearly stated (netiquette, grade 
weighting, models/examples, and 
timing and frequency of 
contributions ). 


Learners have an opportunity to 
get to know the instructor. 


Course contains resources or 
activities mtended to build a sense 
of class community, support open 
communication, and establish trust 
(at least one of the following- 
Icebreaker, Bulletin Board, Meet 
Your Classmates, Ask a Question 
discussion forums). 


Course offers opportunities for 
learner to learner interaction and 
constructive collaboration. 


Assessment & Feedback? 


Course grading policies, including 
consequences of late submissions, 
are clearly stated in the course 
information area or syllabus. 


Course includes frequent and 
appropriate methods to assess 
learners’ mastery of content. 


Criteria for the assessment of a 
graded assignment are clearly 
articulated (rubrics, exemplary 
work). 


Learners have opportunities to 
review their performance and 
assess their own learning 
throughout the course. 


Learners are informed when a 
timed response is required. Proper 
lead time is provided to ensure 


assignments and how quickly to 
expect other regular 
communications (Le., e-mail or 
phone response) from me. 

I establish clear policies and 
expectations for student 
interaction. 


I provide a biography about myself 
or other opportunities for students 
to get to know me. 


Iinclude resources to build a sense 
of community in my course (e.g, 
FAQ page or a discussion topic for 
projects or general questions on 
discussion board). 


I use Internet communication tools 
to promote student interaction and 
collaboration frequently. 


Self-Perception Statement? 


I provide students with an 
explanation of grading policies 
including consequences of late or 
incomplete work. 

I provide multiple opportunities for 
students to demonstrate (assess) 
their learning of the same topics. 


I provide a clear explanation of 
assignment grading criteria to 
students. 


I give students opportunities to 
reflect on their own learning 
throughout the semester, especially 
in the middle and at the end ofa 
semester. 

I give students sufficient time to 


notify me of needed 
accommodations and to complete 
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0.781 


0.813 


0.854 


0.667 


Content Validity Index 
0.990 


0.688 


0.750 


0.917 


0.750 
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there is an opportunity to prepare their assignments. 
an accommodation. 


Leamers have easy access to a I make sure the gradebook is 0.927 
well-designed and up-to-date current and understandable by 

gradebook. students. 

Learners have multiple I give student opportunities to 0.875 
opportunities to provide provide feedback regarding course 

descriptive feedback on course materials, experience, and ease of 

design, course content, course use during the semester. 

experience, and ease of online 

technology. 


Note. Content validity of OSCQR statements as matched with faculty self-perception statements. 
* Indicates subscale items validated for continued study. 

‘Adapted from OSCOQR version 3.0 by SUNY, n.d., Retrieved from http//oscqr.org/. Copyright 
n.d.by the Online Learning Consortium. Adapted courtesy of the copyright holder under a 
Creative Commons License CC by 4.0. (https://creativecommons.org/licenses/by/4.0/) Adapted 
from A study of faculty members’ perceived utilization of best practices in distance learning 
course design and delivery and the role of instructional designers by J. You, 2010, pp. 160—162. 
Retrieved from http//rave.ohiolink.edu/etdc/view?acc_numtoledo1279298347. Adapted with 
permission. 

To accommodate for the gaps in question alignment to OSCQR criterion, the researcher 
limited the scope of this study to only the OSCQR Overview and Information subscale, as this 
was the only complete section of the rubric that could be fully validated with the faculty self 
perception statements. Table 3 lists the self-perception statements aligned with the OSCQR 
Overview and Information subscale and used in this study. 


Table 3 


Faculty Survey Statements Aligned With OSCQR Overview and Information Subscale 


Faculty Survey Statement 
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9. I distribute assignment—task due dates throughout the semester.* 


19. Tinclude an FAQ page or a discussion topic for projects or general questions on discussion 
board.* 


20. luse appropriate terms and icons on the homepage so that course materials can be easily 
located on the course homepage.* 


24. I reference to the university academic dishonesty policy in the syllabus or on the course.* 
29. I use learning objectives to design the student assessment.* 

I provide students with my contact information and office hours.> 

I clearly state on the homepage that the course is in an online format. 

I provide students with a syllabus.> 

I provide course, module, or unit introductions so students know what is expected. 


Note. Adapted from A study of faculty members’ perceived utilization of best practices in 
distance learning course design and delivery and the role of instructional designers by J. You, 
2010, pp. 160-162. Retrieved from http//rave.ohiolink.edwetdc/view?acc_nunFtoledo 
1279298347 Adapted with permission. 
‘Self perception statements and numbering reproduced from You (2010). Selfperception 
statements modified from You (2010) through the content validation process. 
Faculty Self-Perception and OSCQR Results 

Upon completion of the content validation process, data collection was initiated from 
faculty participants. The faculty population was limited to a single small Midwestern health care 
college. The institution had a total of 24 faculty who had taught at least one fully online course in 
the 2018-2019 academic year; these faculty were identified as potential participants. Two of 
these faculty were eliminated from the participant pool because the identified fully online course 
had been cancelled for the academic year, leaving a total population of 22 faculty participants. 
Each faculty was contacted via e-mail recruitment. Of the 22 potential participants, 12 elected to 
participate in the study, resulting in a 55% participation rate. Each faculty participant was 
assigned a participant number used to randomize his or her self-perception survey. A research 


assistant was utilized at this stage of the study to facilitate the randomization. 


Basic demographic information was gathered to garner information about the online 
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teaching experience of faculty respondents. Five faculty participants self-reported having less 
than five years of experience teaching in an online format. Four faculty participants self-reported 
having between five and nine years of experience teaching in an online format. Two faculty 
participants self-reported having between 10 and 14 years of experience. One faculty participant 
reported having more than 15 years’ experience teaching in an online format. When asked to 
identify the number of course design tramings each faculty participant had engaged in during the 
past two years, five faculty participants reported participating in less than five; four faculty 
participants reported participating in five to nine trainings; two faculty participants reported 
participating in 10 to 14 traiings; one faculty participant reported participating i more than 15 
tramings. Recruited faculty participants then responded to an eight-statement survey to gain 
quantitative measurements to ascertain the faculty members’ perceptions of their own course 
design quality with regard to introductory course information. The survey statements asked the 
faculty participants to self-assess and provide a rating on a 4-degree Likert scale. 

Responses from faculty self-perception surveys were aligned to the OSCQR Overview 
and Information subscale rubric rows as identified by the content validation index conducted 
prior to the initiation of this stage of the study (as shown in Table 3). The faculty participant 
identified a course to be reviewed and communicated his or her selection to the research 
assistant. The research assistant assigned the same randomized number to the course that had 
been used for the faculty survey so that the ratings could be matched for analysis. Although some 
of the faculty participants had taught more than one fully online course in the 2018-2019 
academic year, participants each identified a single fully online course to be scored using the 
OSCQR Overview and Information subscale. The courses identified by faculty participants 


included nine undergraduate- and three graduate-level courses. 
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Each course was blindly reviewed by the researcher using the OSCQR Overview and 
Information subscale. The OSCQR subscale ratings were then paired to the faculty self 
perception responses. This matching process allowed the researcher to analyze measures, 
comparing faculty self-perception with observed inclusion of quality course design elements that 
resulted in quantitative data. The data sets were aggregated to determine if alignment existed 
between faculty selfperception of course design quality and the minimum course design quality 
standards the researcher observed using the OSCQR Course Overview and Information subscale. 

Results of the faculty self-perception survey, identified by SP, as matched to the OSCQR 
Overview and Information subscale criterion, identified by CR, are displayed in Table 4. Each 
statement or criterion is followed by an item number indicator utilized in the subsequent data 
analysis tables. 


Table 4 


Participant Response Data 


Randomized Participant Number 


Statement or Criterion 359 6989) 824 9386-5550 899 796 657792 239 «167 = 402 


SP: I use welcome and getting 

started information on the 

homepage so that course 4 4 4 4 4 4 4 4 4 4 4 4 
materials can be easily 

located. (Al) 


CR: Course Includes 
Welcome and Getting Started 4 4 4 4 4 4 3 4 4 4 4 3 
Content. (A2) 


SP: I provide course 

navigation instructions, as 

well as course, module, or 

unit introductions and due 3 4 3, 3 4 4 4 4 4 4 4 4 
dates so students know what 

is expected and how to meet 

the expectations. (B1) 
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CR: An Orientation or 
Overview is provided for the 
course overall, as well as in 
each module. Learners know 
how to navigate and what 
tasks are due. (B2) 


SP: I provide students witha 
viewable and a printable 
syllabus. (Cl) 


CR: A printable syllabus is 
available to learners. (C2) 


SP: I distribute assignment— 
task due dates throughout the 
semester, explain the patterns 
of due dates, and provide easy 
navigation to assignments. 


(D1) 


CR: Course includes a Course 
Information area that 
deconstructs the syllabus for 
learners in a clear and 
navigable way. (D2) 


SP: I reference relevant 
College or University policies 
in the syllabus or in the 
course. (El) 


CR: Course includes links to 
relevant campus policies on 
plagiarism, computer use, 
filing grievances, 
accommodating disabilities, 


etc. (E2) 


SP: I clearly state on the 
homepage that the course is in 
an online format. (F1) 


CR: Course information states 
whether the course is fully 
online, blended, or web- 
enhanced. (F2) 


SP: I use measurable learning 
objectives to design student 
assessments. (Gl) 


CR: Course 
objectives/outcomes are 
clearly defined, measurable, 
and aligned to learning 
activities and assessments. 
(G2) 


33 
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SP: I provide students with 
my contact information and 4 4 2 4 4 4 4 4 4 4 4 4 
office hours. (A1) 


CR: Course provides contact 
information for instructor, 
department, and program. 


(H2) 


SP: How many years have 
you taught in an online <5 <5 59 10-14 59 <5 <5 315 10-14 59 <5 5-9 
format? 


SP: How many different 

course design trainings have 

you participated in over the 5-9 <5 >15 1014 <5 59 <5 59 10-14 59 <5 <5 
past two years? (estimate to 

the best of your ability) 


Note. Table displays self-perception response data and OSCQR score data for individual faculty 
participants. CR = OSCQR Overview and Information subscale criterion; Adapted from OSCOR 
version 3.0 by SUNY, n.d., Retrieved from http://oscqr.org/ Copyright n.d.by the Online 
Learnng Consortium. Adapted courtesy of the copyright holder under a Creative Commons 
License CC by 4.0. (https://creativecommons.org/licenses/by/4.0/). SP = Faculty self-perception 
statement; Adapted from A study of faculty members’ perceived utilization of best practices in 
distance learning course design and delivery and the role of instructional designers by J. You, 
2010, pp. 160-162. Retrieved from http//rave.ohiolink.edwetdc/view?acc_num=toledo 
1279298347 Adapted with permission. 

Results from the faculty self-perception survey were analyzed compared to the OSCQR 
Overview and Information subscale using several analysis methods. One analysis method was 
the Wilcoxon Signed Ranks Test. This nonparametric measurement identifies the differences 
between individual responses of paired data to determine the averages of the differences between 
individual pairs. The Wilcoxon Signed Ranks Test is also used to determine if two dependent 
data sets can be assumed to have similar distributions applicable to all populations. Table 5 
displays the Wilcoxon Signed Ranks Test for the 12 participants. The purpose of this analysis 
was to determine the mean differences between the faculty selfperception and the OSCQR 
Overview and Information subscale rating by individual participant. 


Table 5 


Wilcoxon Signed Ranks Test Table 
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Ttems Number of Occurrences Mean Rank Sum of Ranks 
A2 <Al 2 1.50 3.00 
A2>Al 0 0.00 0.00 
A2=Al 10 

B2 <Bl 6 4.25 25.50 
B2>Bl 1 2.50 2.50 
Bl =Bl 5 

C2<Cl 4 2.50 2.50 
C22 Cl 0 0.00 0.00 
C2=Cl 8 

D2 <D1 11 6.00 66.00 
D2 >D1 0 0.00 0.00 
D2 =D1 1 

E2 <El 4 3.50 14.00 
E2>El 1 1.00 1.00 
E2 =El 7 

F2<Fl 4 3.13 12.50 
F2>Fl 1 2.50 2.50 
F2=F1 7 

G2 <Gl 10 5.50 55.00 
G2 >Gl 0 0.00 0.00 
G2 =Gl 2 

H2 <HI1 sig 6.50 71.50 
H2 >HI 1 6.50 6.50 
H2 =H1 0 

Note. N= 12. 


The data results compared in Table 5 are individual faculty respondents’ self-perception 
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ratings of the inclusion of course design elements within the identified fully online course. Item 
one of each pair marks the faculty respondent rating of self-perception and item two of each pair 
marks the researcher’s rating for the corresponding item as scored using the OSCQR Overview 
and Information subscale. 

The first pair, pair A, is the faculty selfperception rating of the statement: I use welcome 
and getting started information on the homepage so that course materials can be easily located as 
compared to the OSCQR Overview and Information subscale criterion: Course Includes 
Welcome and Getting Started Content. Analyzed results showed that two faculty selfperception 
responses were labeled as negative ranks, which means two of the faculty participants rated 
themselves higher than what was observed by the researcher. Zero faculty selfperception 
responses were labeled as positive ranks, which means that no researcher observations rated the 
observed materials higher than the respondents’ self-ratings. The researcher made observations 
of materials being present that were equal to the faculty selfperceptions of materials inclusion 
for 10 of the faculty self-perception responses, as indicated by tied ranks. 

The mean rank of each comparison identified the average of the items for a particular 
comparison pair. The first pair, pair A, of the faculty selfperception rating of the statement: I use 
welcome and getting started information on the homepage so that course materials can be easily 
located as compared to the OSCQR Overview and Information subscale criterion: Course 
Includes Welcome and Getting Started Content showed that of the two respondents who self 
rated higher than the researcher’s observations, the mean rank was 1.50 and the sum of the 
researcher’s ratings for these two respondents was 3.00. 

Items marked as pair B compared the faculty self-perception statement: I provide course 


navigation instructions, as well as course, module, or unit mtroductions and due dates so student 
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know what is expected and how to meet the expectations with the OSCQR Overview and 
Information subscale criterion: An orientation or Overview is provided for the course overall, as 
well as in each module. Learners know how to navigate and when tasks are due when. The 
Wilcoxon Signed Ranks Test showed that six faculty self-perception responses were labeled as 
negative ranks, which means six of the faculty participants rated themselves higher than what 
was observed by the researcher. One faculty self-perception response was labeled as a positive 
rank, which means that for one faculty respondent the researcher rated the observed materials 
higher than the respondents self-rated. Ties dicated the researcher made observations of 
materials being present that were equal to the faculty self-perceptions of materials inclusion for 
five of the faculty participant responses. The mean rank of Pair B measures showed that six 
respondents self-rated higher than the researcher’s observations, resulting in a mean rank of 4.25 
with a sum of 25.50. The positive rank for this pair showed that of the one faculty participant’ s 
self-perception rating was marked lower than the observations made by the researcher with a 
mean rank of 2.50 and a sum rank of 2.50. 

The next pair, pair C, displayed the faculty self-perception rating of the statement: I 
provide students with a viewable and a printable syllabus as compared to the OSCQR Overview 
and Information subscale criterion: A printable syllabus is available to learners. Results showed 
that four faculty self-perception responses are labeled as negative ranks, which means four of the 
faculty participants rated themselves higher than what was observed by the researcher. Zero 
faculty self-perception responses are labeled as positive ranks, which means that no researcher 
observations were made rating the observed materials higher than the respondents self-rated. 
Ties indicated the researcher made observations of materials being present that were equal to the 


faculty selfperceptions of materials inclusion for eight of the faculty participant responses. The 
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mean rank of each comparison showed that of the four respondents who self-rated higher than 
the researcher’s observations, the mean rank was 2.50 and the sum of the ranks for these four 
participant responses was 10.00. 

The next pair, pair D, displayed the faculty selfperception rating of the statement: I 
distribute assignment—task due dates throughout the semester, explain the patterns of due dates, 
and provide easy navigation to assignments as compared to the OSCQR Overview and 
Information subscale criterion: Course includes a Course Information area that deconstructs the 
syllabus for learnng in aclear and navigable way. Results showed that 11 faculty selfperception 
responses are labeled as negative ranks, which means 11 of the faculty participants rated 
themselves higher than what was observed by the researcher. Zero faculty participant ratings 
qualified for positive ranks, which means that no researcher observations were made rating the 
observed materials higher than the respondents self-rated. As is indicated, the researcher made 
observations of materials being present that were equal to the faculty selfperceptions of 
materials inclusion for one faculty participant’s response. The mean rank of each comparison 
showed that of the 11 respondents who self-rated higher than the researcher’s observations, the 
mean rank was 6.0 and the sum of the ranks was 66.00. 

Items marked as pair E compared the faculty selfperception statement: I reference 
relevant College or University policies in the syllabus or in the course with the OSCQR 
Overview and Information subscale criterion: Course includes links to relevant campus policies 
on plagiarism, computer use, filing grievances, and accommodating disabilities, etc. The 
Wilcoxon Signed Ranks Test showed that four faculty self-perception responses were labeled as 
negative ranks, which means four of the faculty participants rated themselves higher than what 


was observed by the researcher. One faculty selfperception response was labeled as a positive 
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rank, which means that one researcher observation was made rating the observed materials 
higher than the respondent self-rated. Ties indicated the researcher made observations of 
materials being present that were equal to the faculty self-perceptions of materials inclusion for 
seven of the faculty participants’ responses. The mean rank of pair E measures showed that of 
the four respondents who self-rated higher than the researcher’s observations, the mean was 3.50 
and the sum of the faculty participants’ ranks was 14.0. The positive ranks for this pair showed 
that the one faculty participant the researcher scored higher had a mean rank of 1.0 and the sum 
of 1.0. 

The next pair, pair F, displayed the faculty self-perception rating of the statement: I 
clearly state on the homepage that this course is in an online format as compared to the OSCQR 
Overview and Information subscale criterion: Course Information states whether the course is 
fully online, blended, or web-enhanced. Results showed that four faculty selfperception 
responses were labeled as negative ranks, which means four of the faculty participants rated 
themselves higher than what was observed by the researcher. One faculty self-perception 
response was labeled as a positive rank, which means that one researcher observation was made 
rating the observed materials higher than the respondent selfrated. Ties indicated the researcher 
made observations of materials being present that were equal to the faculty selfperceptions of 
materials inclusion for seven faculty participant responses. Of the four respondents who self 
rated higher than the researcher’s observations, the mean rank was 3.13 and the sum of the ranks 
for these four respondents was 12.50. The positive ranks for this pair showed that the one faculty 
participant the researcher scored higher had a mean rank of 2.50 and the sum of those ranks 2.50. 

The next pair, pair G, displayed the faculty self-perception rating of the statement: I use 


measurable learning objectives to design student assessments as compared to the OSCQR 


FACULTY SELF-PERCEPTION OF COURSE DESIGN QUALITY 40 


Overview and Information subscale criterion: Course objectives—outcomes are clearly defined, 
measurable, and aligned to learning activities and assessments. Results showed that 10 faculty 
self-perception responses are labeled as negative ranks, which means 10 of the faculty 
participants rated themselves higher than what was observed by the researcher. No faculty self- 
perception responses are labeled as positive ranks, which means that no researcher observations 
were made rating the observed materials higher than the respondent self-rated. Ties indicated the 
researcher made observations of materials being present that were equal to the faculty self 
perceptions of materials inclusion for two faculty participant responses. The mean rank of each 
comparison showed that of the 10 respondents who self-rated higher than the researcher’s 
observations, the mean rank was 5.50 and the sum of the ranks was 55.00. 

The final pair, pair H, displayed the faculty selfperception rating of the statement: 
provide students with my contact information and office hours.’ as compared to the OSCQR 
Overview and Information subscale criterion: Course provides contact information for instructor, 
department, and program. Results showed that 11 faculty selfperception responses were labeled 
as negative ranks, which means 11 of the faculty participants rated themselves higher than what 
was observed by the researcher. One faculty self-perception response was labeled as a positive 
rank, which means that one researcher observation was made rating the observed materials 
higher than the respondent self-rated. Ties indicated the researcher made no observations of 
materials being present that were equal to the faculty selfperceptions of materials inclusion for 
the faculty participant responses. The mean rank of each comparison showed that of the 11 
respondents who self-rated higher than the researcher’s observations, the mean rank was 6.50 
and the sum of the ranks for these 11 respondents was 71.50. The positive ranks for this pair 


showed that the one faculty participant the researcher scored higher had a mean rank of 6.50 and 
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the sum of those ranks 6.50. 

Summarizing the Wilcoxon Signed Ranks Tests, the researcher identified that the number 
of positive faculty self-perceptions was equal to or greater than the researcher’s positive 
observations in the following pairs of measurements: 

(A1) I use welcome and getting started information on the homepage so that course 

materials can be easily located. 

(A2) Course includes welcome and getting started content. 

(C1) I provide students with a viewable and a printable syllabus. 

(C2) A printable syllabus is available to learners. 

(El) I reference relevant College or University policies in the syllabus or in the course. 

(E2) Course includes Inks to relevant campus policies on plagiarism, computer use, 

filling grievances, accommodating disabilities, etc. 

(F1) I clearly state on the homepage that the course is in an online format. 

(F2) Course information states whether the course is fully online, blended, or web- 

enhanced. 

The Wilcoxon Signed Ranks Test showed that the number of positive faculty self 
perceptions was equal to the researcher’s observations in the following pairs of measurements: 
(B1) I provide course navigation instructions, as well as course, module, or unit 
introductions and due dates so students know what is expected and how to meet the 

expectations. 

(B2) An orientation or overview is provided for the course overall, as well as in each 

module. Learners know how to navigate and what tasks are due. 


The Wilcoxon Signed Ranks Test showed that the number of positive faculty self 
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perception was less than the researcher’s negative observations in the following pairs of 
measurements: 

(D1) I distribute assignment—task due dates throughout the semester, explain the patterns 

of due dates, and provide easy navigation to assignments. 

(D2) Course includes a course information area that deconstructs the syllabus for learners 

in a clear and navigable way. 

(G1) I use measurable learning objectives to design student assessments. 

(G2) Course objectives—outcomes are clearly defined, measurable, and aligned to 

learning activities and assessments. 

(H1) I provide students with my contact information and office hours. 

(H2) Course provides contact information for instructor, department, and program. 

A paired sample t-test was used to determine if a paired set of data resulted in a mean 
differences calculation of zero. A test that results in a zero mean difference indicates that the 
hypothesis of the study is mvalid. Calculation of the paired sample statistics was completed to 
determine if the overall mean of faculty self-perception was similar to or different from the 
OSCQR Overview and Information subscale rating for the group of faculty participants, shown 
in Table 6. 

Table 6 


Paired Sample Statistics 


Pair Item M SD SE 
Al 4.00 0.000 0.000 
Par A 
A2 3.83 0.389 0.122 


Par B Bl 3319 0.452 OAS 
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B2 2.92 1.240 358 

Cl 4.00 0.000 0.000 
Pair C 

C2 3.50 0.905 0.261 

D1 3.58 0.669 0.193 
Pair D 

D2 1.92 0.289 0.083 

El 3.83 0.389 0.112 
Pair E 

E2 3.17 1.267 0.386 

Fl 3.75 0.622 0.179 
Pair F 

F2 3.33 0.888 0.256 

Gl 3.58 0.669 0.193 
Par G 

G2 1.50 1.168 0.337 

Hl 3.83 0.577 0.167 
Pair H 

H2 3.00 0.000 0.000 


Note. N= 12. M= Mean, SD = Standard Deviation, SE = Standard Error Mean 
The Paired Sample Statistics in Table 6 showed that the widest variances (as measured by 
standard deviation) occurred within the researcher’s OSCQR ratings of the courses. The 
following three items are listed in order, starting with the largest standard deviation: 
(E2) Course includes links to relevant campus policies on plagiarism, computer use, 
filing grievances, accommodating disabilities, etc. (SD 1.267) 
(B2) Learners know how to navigate and what tasks are due. (SD 1.240) 
(G2) Course objectives—outcomes are clearly defined, measurable, and aligned to 
learnng activities and assessments. (SD 1.168) 
The widest variance occurred within the researcher’s observations because the researcher has 
been trained to observe the major and mmor differences when observing what is present in 


COUISES. 
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The paired response data was analyzed for correlation between the individual pairs of 
measures. In Table 7, the results of the t-test paired samples correlations are displayed. 
Table 7 


Paired t-Test Samples Correlation 


Pair Correlation Significance 
Par A 0.000 0.000 
Par B O22 0.707* 
Par C 0.000 0.000 
Pair D 0-.196 0.541* 
Par E 0.246 0.441 
Pair F 0.165 0.609* 
Pair G 0.291 0.359 
Par H 0.000 0.000 


Note. N = 12, *p < 0.05 

A significant correlation of greater than 0.05 is noted between the following paired 
faculty self-perception and researcher observation. Pair B items had a significance of 0.707 when 
observing: 

(B1) I provide course navigation instructions, as well as course, module, or unit 
introductions and due dates so students know what is expected and how to meet the 
expectations. 

(B2) An orientation or overview is provided for the course overall, as well as in each 
module. 

Paired D items had a significance of 0.541 when observing: 
(D1) I distribute assignment—task due dates throughout the semester, explain the patterns 


of due dates, and provide easy navigation to assignments. 
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(D2) Course includes a course information area that deconstructs the syllabus for learners 
in a clear and navigable way. 
Paired F items had a significance coefficient of 0.609 when observing: 

(F1) I clearly state on the homepage that the course is in an online format. 

(F2) Course information states whether the course is fully online, blended, or web- 

enhanced. 

No significance was found to be present for the following pairs: 

Par A 

(A1) I use welcome and getting started information on the homepage so that course 

materials can be easily located. 

(A2) Course Includes Welcome and Getting Started Content. 

Pair C 

(C1) I provide students with a viewable and a printable syllabus. 

(C2) A printable syllabus is available to learners. 

Par H 

(H1) I provide students with my contact mformation and office hours. 

(H2) Course provides contact information for instructor, department, and program. 

A t-test of paired samples was also completed to provide analysis of the p-value of the 
pairs of data. A p-value is displayed in Table 8 as the Sig. (two-tailed) indicating that the p-value 
was calculated using the paired data. A Sig. result of 0.005 or larger is considered to be of 
significance. Table 8 displays these values. 

Table 8 


t-Test of Paired Differences 
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Paired Differences 


M SD SE 95% Cl t df Sig. (Two-tailed) 
Pair A 0.167 (0.389 0.112 [-0.081, .0414] 1.483 11 0.166 
PairB —-0.833.—s«(1.267 0.366 0.028, 1.639] 2.278 11 0.044* 
PairC 0.500 ~—0.905 0.261 [-0.075, 1.075] 1.915 11 0.082 
PairD 1.667 _—0.778 0.255  [1.172,2.161] 7.416 11 0.000* 
PairE 0.667. ‘1.231 0.355 [-0.115, 1.449] 1.876 11 0.087 
PairF 0.417 0.996 0.288 [-0.216, 1.050] 1.449 11 0.175 
PairG 2.083 —‘1.165 0.336  [1.343, 2.823] 6.197 11 0.000* 
PairH 0.833. ~—-0.577 0.167  [0.467, 1.200] 5.000 11 0.000* 


Note. M = Mean, SD = Standard Deviation, SE = Standard Error Mean, CI = Confidence Interval, 
* 
‘p <.05. 


Observations from the ¢-test of paired differences demonstrated strong evidence between 
all faculty selfperception measurements and researcher observations with the exception of two 
measures. The faculty selfperception statement (Al): I use welcome and getting started 
information on the homepage so that course materials can be easily located and OSCQR Course 
overview and information criterion item (A2): Course Includes Welcome and Getting Started 
Content showed weak evidence of connection between the faculty selfperception statement and 
the OSCQR Overview and Information subscale measure. The faculty self-perception statement 
(F1): I clearly state on the homepage that the course is in an online format and OSCQR 
Overview and Information subscale item (F2): Course information states whether the course is 
fully online, blended, or web-enhanced also demonstrated weak evidence of connection between 
the faculty selfperception statement and the OSCQR measure. A weak evidence of connection 
means that a large enough difference is present between the faculty self-perception value and the 


OSCQRrubric score based on observations made by the researcher. 
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The results of the study indicated that faculty self-perception of some course materials 
identified by the OSCQR Overview and Information subscale were also observed as present 
when the course materials were reviewed. Additionally, the results indicated that some materials 
are not observed to be present, though the faculty perceived them to be present. Items of 
particular note in this regard are related to supporting students in efficient course navigation and 
outlining expectations faculty have students in the course (related to pair B), helping students to 
understand the syllabus and what work is expected to be completed within a certain timeframe 
(related to pair D), and providing information for students about the delivery method of the 
course (related to pair F). 

Observation of the ¢-test results showed a significant relationship between the following 
faculty selfperception ratings and the corresponding OSCQR Overview and Information 
subscale as displayed in Table 9, demonstrating evidence of a relationship between the faculty 
self-perception values and the researcher observation values of the OSCQR rubric. 

Table 9 


Paired Statements Shown to Be Significant by the t-Test Paired Samples Test 


Faculty Self-Perception Statement* OSCQR Overview & Information Subscale> 
(B1) I provide course navigation instructions, §(B2) An Orientation or Overview is 

as well as course, module, or unit provided for the course overall, as well as in 
introductions and due dates so students know each module. Learners know how to 

what is expected and how to meet the navigate and what tasks are due. 
expectations. 


(C1) I provide students with a viewable and a (C2)A printable syllabus is available to 


printable syllabus. learners. 

(D1) I distribute assignment/task due dates (D2) Course includes a Course Information 
throughout the semester, explain the patterns area that deconstructs the syllabus for 

of due dates, and provide easy navigation to learners in a clear and navigable way. 


assignments. 
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(El) I reference relevant College or (E2) Course includes links to relevant 
University policies in the syllabus or in the campus policies on plagiarism, computer 
course. use, filing grievances, accommodating 


disabilities, etc. 
(G1) I use measurable learning objectives to (G2) Course objectives—outcomes are 
design student assessments. clearly defined, measurable, and aligned to 
learning activities and assessments. 


(H1) I provide students with my contact (H2) Course provides contact information 
information and office hours. for instructor, department, and program 


Note. *Adapted from A study of faculty members’ perceived utilization of best practices in 
distance learning course design and delivery and the role of instructional designers by J. You, 
2010, pp. 160-162. Retrieved from http://rave.ohiolink.edw/etdc/view?acc_num=toledo 
1279298347. Adapted with permission. *Adapted from OSCOR version 3.0 by SUNY, n.d., 
Retrieved from http://oscqr.org/. Copyright n.d.by the Online Learning Consorttum. Adapted 
courtesy of the copyright holder under a Creative Commons License CC by 4.0. 
(https://creativecommons.org/licenses/by/4.0/) 
Summary 

The researcher planned the study to determine if faculty perception of the inclusion of 
course design elements was equivalent to the observed inclusion of course design elements. The 
data collected in this study demonstrated significant findings of some of the faculty self 
perception ratings as compared to some OSCQR overview and information criterion 
observations made by the researcher. The significant findings this study demonstrated the need to 
support faculty in the intentional design of course materials to facilitate student achievement of 
course outcomes and expectations. Although this study focused on course overview and 
information materials, the illuminated gaps support the need for further investigation to 


determine if faculty provide essential materials for students to master the discipline-specific 


content vital to professional success. 
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Chapter 5 
Conclusions, Recommendations, and Summary 

This final chapter serves three purposes: to state conclusions made by the researcher, to 
discuss recommendations the researcher has in response to the study results, and to provide an 
overall summary of the research conducted in this study. 

Conclusions 

Historically, faculty in higher education have been shown to be deficient in specific 
traming related to instructional design, resulting in a lack of intentionality in creating learning 
environments that solicit authentic learnmg (Smidt et al, 2014). Faculty often lack the 
experience and traming to recognize their own gaps in pedagogical practices (You, 2010). When 
faculty neglect to utilize effective teaching strategies or provide necessary communication 
related to course materials, students are unsure of faculty expectations. Therefore, resources are 
necessary to support faculty in the deployment of courses designed to elicit quality learning 
experiences (Rucker et al, 2015). 

The results of this research study support the findings of previous studies in which faculty 
were unable to identify adequately their own gaps in course design quality. One objective of this 
study was to identify faculty perception of their own ability to design courses. This objective was 
expected to be answered through investigation of the question: Do faculty accurately perceive 
their course design abilities? 

Based on paired ¢-test data analysis, the OSCQR Overview and Information subscale was 
compared with faculty selfperceptions in the following broad areas: The presence of materials in 
the course that support efficient student course navigation and communicate general faculty 


expectations of students, the presence of materials that help students to understand clearly the 
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syllabus and what work is expected to be completed within a certain timeftame, and the presence 
of clear information about the delivery method of the course. 

The results of this study indicated that faculty selfperceived they included materials to 
assist students in course welcome and getting started information (Pai A). However, according 
to the observations made by the researcher and scored using the OSCQR Overview and 
Information subscale, some faculty in this study did not provide this information for students. 
Likewise, faculty in this study self-perceived that they communicated the course delivery format 
to students (Pair F). However, the majority of faculty in this study did not clearly communicate 
delivery format to students. 

Liow et al. (1993) determined that in order for students to have the greatest chance of 
achieving all the educational objectives of a course, the course must be designed with 
intentionality and scaffolding in place to support student achievement of the learning goals. 
However, the results of the Wilcoxon Signed Ranks Test within this study demonstrated that 
there are differences between faculty selfperception of their use of learning outcomes to design 
student assessments and the researcher’s observation of defined and measurable learning 
outcomes tied to student assessments (Pair G). 

The results of the t-test for par G showed that the mean paired samples for item G1, 
faculty self-perception, was 3.58 with a standard deviation of 0.669 and the mean paired samples 
for item G2, OSCQR Overview and Information subscale score, was 1.50 with a standard 
deviation of 1.168. The results of the paired differences had a mean of 2.083 with a standard 
deviation of 0.996 with a significant factor of 0.000, indicating this particular pair of items was 
not an accurate measure of the presence of learning objectives in a course if only the statistic 


measure is considered. However, this value may also be an indicator of the faculty self 
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perception of the presence of an item that is not present in the course. These scores demonstrated 
a wide range of observable variance in the connections between learning objectives and student 
assessments. The scores also evidenced faculty self-perception that learning objectives were 
included in student assessments, indicated by the 3.58 mean of faculty selfperception responses. 
Though the significance of the ability to measure accurately this pair is minimal and would need 
to be further tested for reliability. The results pomt to the need for clear communication to 
students about how the learning objectives in a course relate to the course materials and activities 
and the measurability of learning objectives. 

The works of Lea (2004) and Dole and Bloom (2009) identified the need for deliberate 
instructional mteractions, clear mteraction expectations, and formal mstruction related to 
interactions. Thus, the structure of the course must set students up for success in an online 
environment. Using design standards, providing students with a predictable course layout, and 
outlining expected interactions within the learning management system are necessary for 
students to engage with materials at the upper levels of Blooms Taxonomy (Dole & Bloom, 
2009). Consequently, when faculty lack design knowledge, there is a negative impact on the 
student experience in the online environment (Smidt et al, 2014). 

The failure of faculty to recognize clearly and understand their course design assumptions 
in the online environment were validated by the results of pair B in this study. Pair B measured 
faculty self-perception of providing course navigation mstructions, module or unit instructions, 
due dates, and performance expectations as compared to the observed provision of these items 
using the OSCQR Overview and Information subscale criterion. The ¢-test paired sample data 
results showed a faculty self-perception mean of 3.75 with a standard deviation of 0.452 and an 


OSCQR Overview and Information subscale mean of 2.92 with a standard deviation of 1.240. 
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These values showed a wide variance of observed elements, as evidenced by the OSCQR 
standard deviation of 2.92, meaning that observed elements on this OSCQR criterion varied 
nearly 3 points among courses (on the 5-pomt scale), while faculty selfperceived a consistent 

use of the elements in this item. The paired differences of this item showed a mean difference of 
0.833, almost a whole point difference in the means between faculty selfperception and OSCQR 
observations. The standard deviation of this item pair was 1.267, again reflecting a wide variance 
between what is selfperceived and what is observed. The significance of this item pair was 
found to be 0.044, or within the parameters of demonstrating evidence of the measure’s 

accuracy, as the significance ofa two-tailed paired samples t-test determmes significance as 
items approach 0.05. 

Pair C reviewed the faculty self-perception of providing a viewable and printable syllabus 
for students. Pair D reviewed the navigation of assignments, predictability of due dates, and clear 
communication of task due dates for students. Provision of these items would demonstrate 
evidence of predictable and clear navigation for students discussed by Lea (2004) and Dole and 
Bloom (2009). In their 2014 article “Faculty Attitudes about Distance Education,” Smidt et al. 
(2014) investigated the idea that faculty lack knowledge of how to create intentionally a learning 
environment that solicits authentic student learning, how to account naturally for student 
individuality, and how to convey clearly stated outcomes and expectations. The findings 
discussed here further validate the Smidt et al. research related to Pair D in that the majority of 
faculty in this study self-percetved their inclusion of course specific navigation mformation for 
students; however, this information was not recordable in the OSCQR Overview and Information 
subscale. 


Lenert and Janes (2017) called out the need for faculty to nclude course element 
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attributes that are specific to the groups of learners in their courses. The authors specifically 
pointed to the need for faculty to facilitate creation of a student educational community. Progress 
toward the creation of an educational community for students can be achieved by providing 
connections to mstitution-wide policies and procedures, especially with regard to 
accommodations and fully online course performance expectations. Pair E measured faculty self 
perception of the inclusion of campus policies related to plagiarism, computer use, grievances, 
and accommodations as compared to the observations made using the OSCQR Overview and 
Information subscale. The mean of paired differences for item E was 0.667 with a standard 
deviation of 1.231 and significance of 0.087. These results showed a smaller variance between 
faculty selfperception of inclusion of these items in their course as compared to the observed 
items from the OSCQR Overview and Information subscale, but a wider gap between individual 
faculty performances of item inclusion. The strength of the measurement’s significance is 
demonstrated by a small significance as compared to other findings in this study. 

Two additional objectives that the study intended to identify were the faculty strengths 
and weaknesses through a comparison of faculty perception of course design abilities with the 
OSCQR rubric category scores, determined to be limited to the OSCQR Overview and 
Information subscale during the content validation process, and to identify any potential areas for 
suggested faculty trainng in course design standards. These last two objectives were an attempt 
to answer the question: Could training be provided to faculty to address potential gap areas? 

Hixon et al. (2011) completed a study showing that simply providing students with 
course-specific information without guiding them through engagement with the materials does 
not result in an impactful experience for students. Wiggins and McTighe (2008) stated that the 


focus of a course should be on the student and how to guide the student to achieve the desired 
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outcomes of the course using targeted performance measures. However, higher education faculty 
may lack the needed training and experience in implementation of course design methodologies 
that would provide students with value-added experiences in the fully online learning 
environment (Chao et al., 2010). 

Based on the findings of multiple researchers, it was hypothesized that faculty’s self- 
perception of inclusion of basic course design elements could be mismatched with what is 
observable in a deployed online course. Gaps between faculty self-perception and observations 
made using the OSCQR Overview and Information subscale were found in three item pairs using 
the Wilcoxon Signed Ranks Test: Pair D, related to deconstruction of the syllabus and due date 
predictability; Pair G, related to the observation of measurable learning objectives; and Pair H, 
related to contact information for the instructor, program, and department. All three pairs had 
negative ranks that outnumbered the positive or tied ranks. These negative ranks indicated that 
faculty self-perception was scored higher than what is observed using the OSCQR Overview and 
Information subscale for each item. This discrepancy between the two measures means that 
faculty believed materials were included in the course, but those materials were not found when 
the researcher observed the course. 

Limitations and Recommendations 

Based on the study’s limitations and the conclusions drawn in the previous sections, the 
researcher makes the following recommendations: 

Recommendation 1. This study was limited to the OSCQR Overview and Information 
subscale. The researcher recommends expanding the faculty selfperception survey to include 
aligned statements for each of the criterion in the full OSCQR Course Review rubric. To fulfill 


this recommendation, further work would need to be completed to develop self-perception 
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statements and then validate the alignment between the self-perception statements and rubric 
criterion. The researcher was unable to complete a full alignment between the two tools; 
however, achieving alignment between the two tools would provide greater insight into potential 
gaps between the design elements faculty believe they are providing for students and the 
elements observable in a deployed course. 

Recommendation 2. This study was limited to faculty participants from a small 
Midwestern health care college. The faculty participants needed to have taught at least one fully 
online course during a specific academic year. The researcher recommends expanding the 
population sample to a larger set of participants in an attempt to gain transferability of the 
findings across a larger population of online faculty. Expanding the population sample through 
engaging larger institutions of higher education, recruiting ftom multiple institutions, or 
including multiple academic years to determine transferability may eventually lead to 
identification of arecommended training regimen for online faculty in higher education. 

Recommendation 3. This study was limited to group analysis of the data. The researcher 
recommends further analysis of the data to determine if any patterns exist based on the number 
of years of online teaching experience or the amount of course design traning completed by the 
participants. The existence of any identifiable patterns could inform a suggested traming regimen 
for online faculty, especially in this particular institution. 

Recommendation 4. This study identified specific gaps in faculty self-perception of the 
distribution of assignments and tasks throughout the term, the outward alignment of learning 
outcomes to guide assignment expectations, and the presence of contact information for the 
instructor, departments, and program. The researcher recommends providing targeted traming 


and written guidelines for this particular institution to facilitate consistent guidelines for faculty 
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inclusion of these course design elements within all fully online courses. 
Summary 

This study concluded with limited success. The research started out to answer specific 
questions related to course design quality. These essential questions were: 

e Do faculty accurately perceive their online course design abilities? 

e@ What gaps exist between faculty’s self-perception of online course design and 

implementation of essential course design elements in their courses? 

e Could training be provided to faculty to address potential gap areas? 

The study intended to look at all elements of course design within a full course. However, the 
researcher was unable to validate alignment between all of the faculty selfperception questions 
in the You (2010) study, A Study of Faculty Members’ Perceived Utilization of Best Practices in 
Distance Learning Course Design and Delivery and the Role of Instructional Designers, and the 
full OSCQR rubric. 

The initial study intention of reviewing full course design element implication was not 
achieved. However, partial review of course design elements in the OSCQR Overview and 
Information subscale provided nformation demonstrating gaps between faculty selfperception 
of the inclusion of course design elements and the course design elements observed in the course. 
Therefore, the researcher was able to answer partially the essential questions identified at the 
onset of the study. The research was able to identify four recommendations for further study. 
These additional recommendations were targeted to support additional validation of the results of 


this study and to affirm the historical studies used to support this study’s research. 
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APPENDIX A 
Faculty Self Perception Statements 
The following list of 30 faculty selfperception statements is from You (2010): “Please specify 
the extent that you agree that your distance learning courses reflect the following best practices 


of design and delivery. Select the one that best matches your opinion. 


a. Strongly disagree b. Disagree  c. Agree d. Strongly agree 


1. I use Internet communication tools to promote faculty and student interaction frequently. 
2.1 establish clear policies for faculty-student communication. 

3. I use Internet communication tools to encourage student-student interaction. 

4. I make student’s thinking visible to entire class (ask students to post their thoughts, 
homework, or group projects on class discussion board or blog, wiki). 

5. I use active learning activities (e.g. discussions, case studies, problem-based learning, and 
simulations etc.) to engage students. 

6. I use simulated or real-world problems to engage students’ learning. 

7. provide constructive feedback on student’ assignments and other inquiries in a timely 
manner. 

8. I give students sufficient time to complete their assignments. 

*Q_ I distribute assignment/task due dates throughout the semester. 

10. I assign challenging tasks to students to communicate high expectations. 

11. I select exemplary student projects and make them available as examples to the class with 


student’s permission. 
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12. Tallow students choose their own projects according to the project requirements with my 
guidance. 

13. Luse different media formats to present course materials (e.g., web pages, audio, or video 
clips). 

14. Iconsider students’ learning characteristics in the selection of technological tools. 

15. I give students opportunities to reflect on their own learning throughout the semester, 
especially in the middle and at the end of a semester. 

16. I give student opportunities to provide feedback regarding course content during the 
semester. 

17. Imake specific course resources (professional journals, associations, user groups) available 
to students on the course site. 

18. I add links to the library, tech support, writing support, etc on the course site. 

*19.T include an FAQ page or a discussion topic for projects or general questions on discussion 
board. 

*20. luse appropriate terms and icons on the homepage so that course materials can be easily 
located on the course homepage. 

21. L organize the course materials in a logical format. 

22. I use visual aids when necessary to support student learning. 

23. I work with instructional designers to make sure that students with special needs can access 
course materials. 

*24. I reference to the university academic dishonesty policy in the syllabus or on the course. 
25.1 used assessment strategies that deter academic dishonesty (e.g. use authentic assessment, 


proctored tests, lockdown browser, or Turnitinn etc.) 
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26. I follow the fair use guidelines (or TEACH Act) when using copyrighted materials. 

27.1 use learning objectives to design the learning activities. 

28. luse learning objectives in the selection of technological tools. 

*29. IT use learning objectives to design the student assessment. 

30. What other strategies or best practices of distance learning course design and delivery you 
use in your online courses but not listed in this survey?” 

(You, 2010, p. 160-162). *Indicates items used in this study. 

Optional Demographic Questions: 


1. Indicate the number of fully online courses you have taught for any College or 


Unwersity. 
a. 0-4 
b. 5-9 
c. 10-14 


d. 15 or more 


2. Indicate the number of fully online courses you have design for any College or 


University. 
a. 0-4 
b. 5-9 
c. 10-14 
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